Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cluster template for vsphere #1129

Merged
merged 6 commits into from
Oct 24, 2023
Merged

cluster template for vsphere #1129

merged 6 commits into from
Oct 24, 2023

Conversation

jkremser
Copy link
Contributor

@jkremser jkremser commented Oct 13, 2023 •
edited
Loading

Issue: giantswarm/roadmap#2191

What does this PR do?

Updating the old templates that don't work with the current capv provider to new ones that use cluster app instead. This is needed for auto IP allocation for workload cluster's api server. We use the ipam controller for that and IPAddressClaim CR is created by cluster helm chart.

./kubectl-gs template cluster --provider vsphere \
   --organization giantswarm \
   --name gstempl1 \
   --vsphere-service-load-balancer-cidr 10.10.222.238/32 | k apply -f -


# 3 minutes later..

λ k get vspheremachine -A | grep gstempl                                                                                                                                           
org-giantswarm   gstempl1-control-plane-05c119fe-f5tgr     gstempl1                                                              47s
org-giantswarm   gstempl1-control-plane-05c119fe-jlrxz     gstempl1     true    vsphere://4208a5f0-9f06-1c6a-c027-fb604dc6c388   3m37s
org-giantswarm   gstempl1-default-4418a143-9ttrk           gstempl1     true    vsphere://4208d914-3acc-5d7f-3fcd-247947a569f7   3m41s
org-giantswarm   gstempl1-default-4418a143-bdqq8           gstempl1     true    vsphere://4208ad8b-99ab-9334-ff8e-c63f9320d934   3m40s
org-giantswarm   gstempl1-default-4418a143-tmqpz           gstempl1     true    vsphere://42084f88-c4c0-7891-ecbd-52d27c529068   3m40s

λ govc ls /Datacenter/vm/ | grep gstempl                                                                                                                                                                   

/Datacenter/vm/gstempl1-gw4j5
/Datacenter/vm/gstempl1-worker-84d565588xb6n95-tm2k7
/Datacenter/vm/gstempl1-worker-84d565588xb6n95-gvprp
/Datacenter/vm/gstempl1-ts49x
/Datacenter/vm/gstempl1-worker-84d565588xb6n95-vh9mj

# 7 minutes later..

λ k tree clusters -n org-giantswarm gstempl1  
NAMESPACE       NAME                                                           READY  REASON  AGE
org-giantswarm  Cluster/gstempl1                                               True           7m1s
org-giantswarm  ├─AdmissionReport/152bdc98-61f5-4801-b0e4-cc1232818642         -              7m1s
org-giantswarm  ├─BackgroundScanReport/152bdc98-61f5-4801-b0e4-cc1232818642    -              6m30s
org-giantswarm  ├─KubeadmConfigTemplate/gstempl1-worker-49eeb741               -              7m1s
org-giantswarm  ├─KubeadmControlPlane/gstempl1                                 True           7m1s
org-giantswarm  │ ├─Machine/gstempl1-2pqt2                                     True           86s
org-giantswarm  │ │ ├─KubeadmConfig/gstempl1-kjc8p                             True           86s
org-giantswarm  │ │ │ └─Secret/gstempl1-kjc8p                                  -              86s
org-giantswarm  │ │ └─VSphereMachine/gstempl1-control-plane-05c119fe-845gn     True           86s
org-giantswarm  │ │   └─VSphereVM/gstempl1-2pqt2                               True           85s
org-giantswarm  │ ├─Machine/gstempl1-gw4j5                                     True           3m46s
org-giantswarm  │ │ ├─KubeadmConfig/gstempl1-nwjff                             True           3m46s
org-giantswarm  │ │ │ └─Secret/gstempl1-nwjff                                  -              3m46s
org-giantswarm  │ │ └─VSphereMachine/gstempl1-control-plane-05c119fe-f5tgr     True           3m46s
org-giantswarm  │ │   └─VSphereVM/gstempl1-gw4j5                               True           3m46s
org-giantswarm  │ ├─Machine/gstempl1-ts49x                                     True           6m36s
org-giantswarm  │ │ ├─KubeadmConfig/gstempl1-8jl25                             True           6m36s
org-giantswarm  │ │ │ └─Secret/gstempl1-8jl25                                  -              6m36s
org-giantswarm  │ │ └─VSphereMachine/gstempl1-control-plane-05c119fe-jlrxz     True           6m36s
org-giantswarm  │ │   └─VSphereVM/gstempl1-ts49x                               True           6m35s
org-giantswarm  │ ├─Secret/gstempl1-ca                                         -              6m36s
org-giantswarm  │ ├─Secret/gstempl1-etcd                                       -              6m36s
org-giantswarm  │ ├─Secret/gstempl1-kubeconfig                                 -              6m36s
org-giantswarm  │ ├─Secret/gstempl1-proxy                                      -              6m36s
org-giantswarm  │ └─Secret/gstempl1-sa                                         -              6m36s
org-giantswarm  ├─MachineDeployment/gstempl1-worker                            True           7m1s
org-giantswarm  │ ├─AdmissionReport/da264148-fc21-42ba-8124-155d0a6acc81       -              7m1s
org-giantswarm  │ ├─BackgroundScanReport/da264148-fc21-42ba-8124-155d0a6acc81  -              6m29s
org-giantswarm  │ └─MachineSet/gstempl1-worker-84d565588xb6n95                 True           6m40s
org-giantswarm  │   ├─Machine/gstempl1-worker-84d565588xb6n95-gvprp            True           6m39s
org-giantswarm  │   │ ├─KubeadmConfig/gstempl1-worker-49eeb741-cmz7b           True           6m39s
org-giantswarm  │   │ │ └─Secret/gstempl1-worker-49eeb741-cmz7b                -              5m43s
org-giantswarm  │   │ └─VSphereMachine/gstempl1-default-4418a143-bdqq8         True           6m39s
org-giantswarm  │   │   └─VSphereVM/gstempl1-worker-84d565588xb6n95-gvprp      True           5m43s
org-giantswarm  │   ├─Machine/gstempl1-worker-84d565588xb6n95-tm2k7            True           6m39s
org-giantswarm  │   │ ├─KubeadmConfig/gstempl1-worker-49eeb741-9cttg           True           6m39s
org-giantswarm  │   │ │ └─Secret/gstempl1-worker-49eeb741-9cttg                -              5m43s
org-giantswarm  │   │ └─VSphereMachine/gstempl1-default-4418a143-tmqpz         True           6m39s
org-giantswarm  │   │   └─VSphereVM/gstempl1-worker-84d565588xb6n95-tm2k7      True           5m43s
org-giantswarm  │   └─Machine/gstempl1-worker-84d565588xb6n95-vh9mj            True           6m40s
org-giantswarm  │     ├─KubeadmConfig/gstempl1-worker-49eeb741-trd8d           True           6m40s
org-giantswarm  │     │ └─Secret/gstempl1-worker-49eeb741-trd8d                -              5m43s
org-giantswarm  │     └─VSphereMachine/gstempl1-default-4418a143-9ttrk         True           6m40s
org-giantswarm  │       └─VSphereVM/gstempl1-worker-84d565588xb6n95-vh9mj      True           5m43s
org-giantswarm  ├─MachineHealthCheck/gstempl1                                  -              7m1s
org-giantswarm  ├─VSphereCluster/gstempl1                                      True           7m1s
org-giantswarm  │ ├─AdmissionReport/1117c96f-4fd2-4b91-adb4-b5f4d9337576       -              7m1s
org-giantswarm  │ ├─BackgroundScanReport/1117c96f-4fd2-4b91-adb4-b5f4d9337576  -              6m29s
org-giantswarm  │ ├─Secret/gstempl1-credentials                                -              7m1s
org-giantswarm  │ ├─VSphereMachine/gstempl1-control-plane-05c119fe-845gn       True           86s
org-giantswarm  │ │ └─VSphereVM/gstempl1-2pqt2                                 True           85s
org-giantswarm  │ ├─VSphereMachine/gstempl1-control-plane-05c119fe-f5tgr       True           3m46s
org-giantswarm  │ │ └─VSphereVM/gstempl1-gw4j5                                 True           3m46s
org-giantswarm  │ ├─VSphereMachine/gstempl1-control-plane-05c119fe-jlrxz       True           6m36s
org-giantswarm  │ │ └─VSphereVM/gstempl1-ts49x                                 True           6m35s
org-giantswarm  │ ├─VSphereMachine/gstempl1-default-4418a143-9ttrk             True           6m40s
org-giantswarm  │ │ └─VSphereVM/gstempl1-worker-84d565588xb6n95-vh9mj          True           5m43s
org-giantswarm  │ ├─VSphereMachine/gstempl1-default-4418a143-bdqq8             True           6m39s
org-giantswarm  │ │ └─VSphereVM/gstempl1-worker-84d565588xb6n95-gvprp          True           5m43s
org-giantswarm  │ └─VSphereMachine/gstempl1-default-4418a143-tmqpz             True           6m39s
org-giantswarm  │   └─VSphereVM/gstempl1-worker-84d565588xb6n95-tm2k7          True           5m43s
org-giantswarm  ├─VSphereMachineTemplate/gstempl1-control-plane-05c119fe       -              7m1s
org-giantswarm  └─VSphereMachineTemplate/gstempl1-default-4418a143             -              7m1s

λ k get ipaddress -A
NAMESPACE        NAME       ADDRESS         POOL NAME   POOL KIND
org-giantswarm   gstempl1   10.10.222.232   wc-cp-ips   GlobalInClusterIPPool

λ k get globalinclusterippools.ipam.cluster.x-k8s.io 
NAME        ADDRESSES                         TOTAL   FREE   USED
wc-cp-ips   ["10.10.222.232-10.10.222.239"]   8       7      1

Should this change be mentioned in the release notes?

  • CHANGELOG.md has been updated

@jkremser jkremser requested a review from a team as a code owner October 13, 2023 10:13
@jkremser jkremser force-pushed the update-vsphere branch 3 times, most recently from e49b640 to adb5f6a Compare October 13, 2023 10:30
@jkremser jkremser requested a review from a team October 23, 2023 09:19
@jkremser
cluster template for vsphere
976bef9 
Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>
@jkremser
CVE-2023-39325 > nancy-ignore
367df1d 
Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>
@jkremser jkremser force-pushed the update-vsphere branch 2 times, most recently from 1d4150b to c0be633 Compare October 23, 2023 11:31
@vxav
Copy link
Contributor

vxav commented Oct 23, 2023

I don't see ipPoolName or networkName, we probably need them?

Out of scope:
Also the template is common to worker and control plane.
This made me think that we never had a need to run different images. Do we even need to allow it in the cluster chart interface?

@jkremser
add unit test
7a3669a 
Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>
@jkremser
Copy link
Contributor Author

jkremser commented Oct 23, 2023 •
edited
Loading

networkName:

flagVSphereNetworkName             = "vsphere-network-name"

so --vsphere-network-name=foo should do.

ipPoolName: yep this one is missing, I can add it
edit: should be there (651bf8e)

@jkremser
capv: add configurable ip pool name
651bf8e 
Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>
@erkanerol
Copy link
Contributor

erkanerol commented Oct 23, 2023 •
edited
Loading

Looks Good To Me in general. We can add/change flags to improve user experience later. I compared how it is implemented and saw it is the same as other CAPI providers, which is really nice. Thank you!

Please don't merge without getting review from honeybadger :)

@jkremser
capv: Better logic for default k8s version
92d5379 
Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>
@vxav
Copy link
Contributor

vxav commented Oct 24, 2023

Very nice indeed, this will be very useful !

@uvegla
Copy link
Contributor

uvegla commented Oct 24, 2023 •
edited
Loading

If not too much trouble, could you please add support for this in gitops add base command:

kubectl-gs/cmd/gitops/add/base/runner.go

Line 95 in 92d5379

func generateClusterBaseTemplates(config common.StructureConfig) (common.ClusterBaseTemplates, error) {

I don't know how often gitops add base is used, it is just that it would be nice to have.

@jkremser
capv: add support also for 'k gs gitops add base --provider vsphere'
1764739 
Signed-off-by: Jirka Kremser <jiri.kremser@gmail.com>
@jkremser
Copy link
Contributor Author

@uvegla done in 1764739

@jkremser jkremser merged commit 1d04f4b into main Oct 24, 2023
1 check passed
@jkremser jkremser deleted the update-vsphere branch October 24, 2023 12:21
@jkremser jkremser mentioned this pull request Oct 24, 2023
Closed
@uvegla
Copy link
Contributor

uvegla commented Oct 24, 2023

@jkremser Thank you! ❤️

vxav
vxav reviewed Oct 25, 2023
View reviewed changes
cmd/template/cluster/flag.go
// VSphere only
cmd.Flags().StringVar(&f.VSphere.ControlPlane.IP, flagVSphereControlPlaneIP, "", "Control plane IP, leave empty for auto allocation.")
cmd.Flags().StringVar(&f.VSphere.ServiceLoadBalancerCIDR, flagVSphereServiceLoadBalancerCIDR, "", "CIDR for Service LB for new cluster")
cmd.Flags().StringVar(&f.VSphere.NetworkName, flagVSphereNetworkName, "grasshopper-capv", "Network name in vcenter that should be used for the new VMs")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't pick it up but we should probably remove grasshopper-capv as a default value 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vxav vxav vxav approved these changes

@erkanerol erkanerol erkanerol approved these changes

@uvegla uvegla uvegla approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jkremser @vxav @erkanerol @uvegla